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LONG-TERM  GOALS 

The  Macaulay  Library  is  the  world’s  largest  archive  of  animal  sounds  and  has  been  selected  by  the 
Office  of  Naval  Research  as  a  major  repository  for  the  deposition,  digital  archival,  review,  and 
retrieval  of  the  many  recordings  of  marine  animals  made  over  the  last  half-century.  Archived  marine 
recordings  pose  challenging  retrieval  problems  given  the  typically  long  intervals  of  silence  between 
animal  sounds  and  the  multiplicity  of  species  detectable  in  any  given  recording.  One  goal  of  this 
project  is  to  design  software  that  will  permit  remote  experts  to  annotate  the  content  of  long  recordings 
archived  at  the  Macaulay  Library  through  their  web  browsers.  Annotations  will  permit  subsequent 
searches  of  the  archive  database  to  retrieve  not  only  suitable  recordings,  but  also  those  parts  of  a 
recording  meeting  the  search  criteria.  The  project  also  seeks  to  define  and  extract  a  set  of  acoustic 
features  from  all  archived  marine  recordings  that  can  be  used  in  subsequent  search  and  retrieval  tasks. 
Both  capabilities  will  be  unique  to  this  sound  collection,  and  will  greatly  enhance  the  accessibility  and 
the  utility  of  the  archive  to  scientists,  students,  educators,  military  personnel,  and  the  media. 

OBJECTIVES 

To  achieve  the  annotation  goals,  the  project  must  provide  a)  a  browser-based  software  tool  for 
visualizing  and  playing  back  digitized  sounds  stored  in  the  archive;  b)  mouse-driven  tools  for 
identifying  specific  segments  within  the  visual  image  of  the  sound;  c)  pull  down  menus  that  allow  the 
annotator  to  assign  standardized  metadata  terms  for  annotation  as  well  as  entry  of  custom  notes  to 
specific  segments  within  recordings;  d)  suitable  metadata  structures  for  storage  of  the  annotations  and 
the  relevant  segment  delimitation  points  and  linkages  to  other  relevant  metadata  fields;  e)  search 
engines  that  support  the  invocation  of  annotation  terms  during  searches  along  with  other  standard 
criteria;  and  f)  retrieval  tools  that  identify  relevant  parts  within  archived  recordings,  mount  the 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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recording,  create  a  visual  image,  and  move  to  the  relevant  annotated  segments.  All  of  these  tools  must 
allow  multiple  annotators  to  add  infonnation  to  the  same  recording,  and  search  engines  that  permit 
either  all-annotator  or  specific-annotator  searches.  To  our  knowledge,  no  archive  currently  provides 
suitable  tools  to  meet  these  requirements.  They  thus  must  be  created  from  scratch  by  this  project. 

To  achieve  the  extracted  feature  goal,  we  need  to:  a)  obtain  a  consensus  from  the  marine  acoustics 
community  on  which  common  measures  would  best  facilitate  their  use  of  the  sound  archive;  b)  create 
the  relevant  algorithms  and  test  them;  c)  implement  the  algorithms  so  that  they  can  be  applied  directly 
to  annotated  segments  in  the  archive;  and  d)  provide  suitable  metadata  structures  to  store  the  extracted 
feature  data  and  link  them  to  the  other  fields  pertinent  to  any  recording.  At  some  later  date,  automatic 
batch  signal  detectors  and  segment  delimiters  will  be  developed  that  will  allow  extraction  on  all 
recordings,  whether  yet  annotated  or  not.  However,  that  step  is  not  part  of  the  current  project. 

APPROACH  AND  WORK  PLAN 

The  annotation  and  feature  extraction  software  tools  are  both  complex  tasks  because  they  require 
seamless  integration  of  visualization,  user  input,  database  design  and  manipulation,  search  engine,  and 
web  usability  components.  The  tasks  have  been  broken  into  discrete  modules  and  specifications  written 
for  each  module,  including  definition  of  deliverables  and  due  dates.  The  Macaulay  Library  metadata 
model  required  a  major  reconstruction  and  transfer  into  an  Oracle  relational  enviromnent.  Critical  to 
this  effort  was  the  incorporation  of  existing  or  development  of  new  metadata  standards  for  habitat  type, 
taxonomy  resolution,  and  classification  of  associated  behaviors.  All  software  for  search,  retrieval,  and 
manipulation  is  being  written  in  Java  to  ensure  cross-platform  compatibility.  The  visualization  tool, 
(creating  spectrograms  and  waveforms  of  any  archived  sound),  posed  unique  challenges  because  it  had 
to  work  in  a  browser  environment,  handle  both  audio  and  video  archived  files,  guarantee  proper  digital 
rights  protection,  be  instantaneous  in  operation,  allow  for  segment  selection  by  remote  users,  and  meet 
the  high  standards  for  spectrogram  resolution  demanded  by  the  marine  bioacoustics  community.  Our 
solution  has  been  to  build  this  tool  using  Apple  Quicktime,  a  utility  available  on  all  user  platforms  for 
free  and  providing  all  the  necessary  requirements.  The  feature  extraction  tools  require  input  from  the 
marine  bioacoustics  community  at  all  stages.  This  is  being  handled  by  a)  including  critical  members  of 
that  community  as  partners  in  the  project,  b)  holding  workshops  where  optimal  features  can  be 
discussed,  and  c)  recruiting  testing  and  critique  input  from  a  variety  of  marine  bioacoustics  labs 
throughout  the  United  States  as  algorithms  are  implemented. 

Senior  programmer  William  Sandner  oversees  the  project,  delegates  JAVA  tasks,  and  works  himself 
on  critical  parts  of  the  module.  Rafe  Rosen  and  Gui  Iacino  each  work  on  programming  other  modules 
in  the  project.  A  number  of  Cornell  undergraduates  in  Computer  Science  have  also  contributed  to 
programming  tasks  under  Mr.  Sandner’ s  direction.  Erica  Olsen  is  our  web  interface  designer  and 
usability  tester.  Tim  Levatich  handles  all  the  metadata  modeling,  implementation,  and  database 
management.  Roger  Slothower  is  developing  the  mapping  tools.  The  Quicktime  visualization  tool  was 
developed  by  Totally  Hip  Technologies,  Inc.,  a  Vancouver,  British  Columbia  company  that  specializes 
in  Quicktime  extensions.  The  Bioacoustics  Research  Program  at  the  Cornell  Lab  of  Ornithology, 
including  partners  Dr.  Chris  Clark  and  Kurt  Fristrup,  are  providing  critical  input  on  the  optimal  design 
of  the  visualization  tools  and  the  feature  extraction  module.  Partner  Dr.  David  Mellinger  has  also 
provided  critical  input  on  feature  selection,  and  will  implement  the  initial  algorithms  in  a  Matlab 
environment  so  that  they  can  be  distributed,  examined,  and  critiqued  by  participating  marine 
bioacoustics  laboratories  around  the  country.  Dr.  Sue  Moore,  another  partner,  will  be  providing  critical 
assistance  in  recruiting  suitable  participant  labs  and  in  feature  selection. 
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As  noted  below,  the  search,  playback,  and  visualization  tools  are  completed  and  available  for  beta 
testing  on  the  Macaulay  Library  website.  The  digitized  marine  animal  sound  collection  is  also  now 
available  for  browsing.  The  goal  for  the  remainder  of  the  year  is  to  complete  the  annotator  interface, 
test  it  with  volunteer  annotators,  and  refine  its  usability.  The  feature  extraction  algorithms  have  been 
selected  and  written  into  Matlab.  The  goal  this  year  is  to  obtain  testing  of  them  in  various  research 
labs,  refine  them,  and  then  implement  them  so  that  all  annotated  segments  of  recorded  sounds  are 
concurrently  feature  extracted  in  the  background.  Search  tools  using  the  stored  features  will  also  be 
completed  this  coming  year. 

WORK  COMPLETED 

The  revision  of  the  database  and  importation  of  legacy  data  into  the  new  Oracle  environment  was 
completed  in  the  first  year  of  the  award.  This  was  followed  in  April  2005  with  the  initial  release  of  the 
search,  search  results,  mapping  and  sound  visualization  tools.  The  latter  have  been  significantly  refined 
in  the  ensuing  8  months.  A  typical  search  results  page  now  appears  as  follows: 


Page  1  of  29  -  257  results 
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Figure  1:  Sample  results  page  after  search  for  “humpback  whale” 


Once  a  search  is  completed,  the  user  can  then  use  the  now  widely  familiar  Google  map  tools  to  display 
the  locations  of  the  recordings  found  by  the  search.  Clicking  on  any  of  the  site  tags  creates  a  balloon 
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providing  some  information  about  that  site’s  recordings  and  allowing  the  user  to  play  that  sound  and/or 
obtain  more  information  about  the  recording: 
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Figure  2:  Google  map  display  of  locations  of  recordings  and  balloon  giving  more  information 

and  playback  option  for  a  selected  site. 


The  user  can  then  play  a  selected  recording  from  either  the  map  or  search  results  and  visualize  the 
sounds  as  waveforms  and/or  spectrograms  using  a  variety  of  customizable  settings  for  bandwidth,  time 
scale,  frequency  scale,  color,  and  gridding.  If  the  recording  is  a  video,  the  player  will  also  show  the 
video  synchronized  with  the  sound  and  visualization  displays.  The  player  is  based  on  Quicktime, 
which  is  free  to  all  users  and  operates  on  both  Windows  and  Macintosh  platforms.  The  necessary  plug¬ 
in  is  available  on  the  Macaulay  Library  web  site.  Once  installed,  the  user  can  use  the  player  to  play  and 
visualize  recordings  found  in  a  Macaulay  Library  search  or  recordings  stored  on  their  own  computer. 
This  allows  users  to  compare  a  new  recording  of  their  own  to  reference  recordings  in  the  archive. 
Customized  settings  for  spectrogram  and  wavefonn  views  can  be  stored  on  the  user’s  computer  to 
facilitate  rapid  standardization  of  the  displays.  The  user  can  also  identify  specific  sections  within  a 
recording  at  which  they  want  playing  to  begin  and  end.  The  ability  to  insert  these  “mark  in”  and  “mark 
out”  points  will  also  be  used  by  annotators  when  the  annotation  tools  are  complete.  A  sample  page 
from  the  player  appears  below: 
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Figure  3:  Sample  page  from  free  Macaulay  Library  player  showing  waveform,  spectrogram, 
and  control  setting  tabs  for  an  audio  recording  of  a  humpback  whale. 

Selection  of  suitable  feature  measurements  to  be  made  automatically  on  all  annotated  segments  of 
audio  files  is  complete,  and  the  relevant  algorithms  have  been  programmed  into  Matlab  modules  for 
distribution  to  participating  laboratories  for  testing.  Some  substitutions  or  additions  of  features  may 
occur  based  on  feedback  from  the  marine  bioacoustics  community. 

RESULTS 

Major  results  from  this  project  to  date  include  the  creation  of  a  proposed  new  metadata  standard  for  the 
classification  of  animal  behavior  (http://ethodata.comm.nsdl.org/),  an  online  tool  produced  in 
collaboration  between  the  Cornell  Lab  of  Ornithology  and  uBio  (http://www.ubio.org/)  for  resolution 
of  taxonomic  synonymies,  and  the  development  of  a  browser-compatible  platform-independent  sound 
visualization  tool  that  includes  appropriate  digital  rights  protections  and  the  ability  to  identify 
segments  within  recordings  for  annotation  and  segment  relocation  and  distribution. 

IMPACT  AND  APPLICATIONS 
National  Security 

While  the  U.S.  Navy  has  its  own  archives  of  marine  animal  sounds  that  can  be  used  to  discriminate 
between  man-made  devices  and  animals,  the  growing  Macaulay  Library  archive  is  becoming  a  broader 
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and  more  definitive  source  as  it  grows  and  users  become  more  adept  at  exploiting  the  new  tools  to 
identify  and  retrieve  specific  sounds.  The  military  will  of  course  have  full  access  to  the  entire 
collection. 

Economic  Development 

Both  commercial  and  public  media  frequently  seek  examples  of  sounds  of  newsworthy  species  from 
the  Macaulay  Library.  The  new  tools  developed  here  will  make  such  requests  much  faster  and  effective 
as  the  media  themselves  will  be  able  to  search,  audition,  and  select  specific  segments  of  audio  and 
video  recordings  remotely  on  the  web  and  download  them  for  usage  immediately. 

Quality  of  Life 

The  general  public  is  largely  fascinated  by  marine  mammals.  Free  access  to  our  videos  and  audio 
recordings  of  these  animals  can  enrich  the  lives  of  anyone  with  access  to  an  online  computer  and 
browser. 

Science  Education  and  Communication 

With  support  from  NSF,  the  Macaulay  Library  is  working  with  teachers  to  use  its  archives  of  animal 
behavior  to  enrich  STEM  education.  The  natural  allure  of  marine  animals  provides  a  superb 
springboard  to  introduce  students  to  many  topics  in  biology  and  physics,  as  well  as  applied  areas  such 
as  conservation  and  environmental  change. 

TRANSITIONS 
Economic  Development 

The  Macaulay  Library  is  a  major  resource  of  sound  and  video  recordings  for  public  and  commercial 
media,  museums,  zoos  and  aquaria,  producers  of  products  reproducing  animal  sounds,  wildlife 
identification  devices,  CD  and  DVD  nature  productions,  sound  effects  for  the  movie  industry,  etc. 

Quality  of  Life 

Macaulay  Library  provides  sounds  for  web-based  and  kiosk  public  information  sites,  training  tapes  for 
bioacoustic  censusing  by  wildlife  biologists  and  conservation  staff,  workshops  to  train  wildlife 
recordists  and  bioacoustic  censusing  staff,  etc. 

Science  Education  and  Communication 

Macaulay  Library  is  one  of  the  world’s  primary  resources  for  archived  sounds  of  animals  and  is  thus 
widely  used  by  scientists,  teachers,  and  students.  It  is  also  an  active  member  of  the  National  Science 
Digital  Library  (NSDL)  program  and  its  collections  are  available  through  NSDL  portals. 

RELATED  PROJECTS 

Macaulay  Library  also  has  a  concurrent  grant  from  the  NSF-funded  National  Science  Digital  Library 
program  to  maintain  an  NSDL  portal  and  web  pages  that  will  maximize  access  and  utilization  of  the 
animal  sound  and  video  collections  for  education  at  all  levels.  It  has  another  (smaller)  NSF  grant  to 
fund  the  development  of  an  international  metadata  standard  for  the  field  of  animal  behavior.  Finally, 
ONR  is  supporting  generously  the  archival  of  the  last  half-century’s  accumulated  field  recordings  of 
marine  animals  at  the  Macaulay  Library.  This  has  included  funds  for  both  equipment  acquisition  and 
staff  salaries  for  archivists. 
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